1. Field of the Invention
The present invention relates to an image processing apparatus, an image capturing apparatus, an image processing method, and a storage medium for obtaining information about focus control of a subject.
2. Description of the Related Art
Conventionally, in the case where an image was captured by a camera out of focus, it was necessary to recapture the image after performing focusing again. Further, in the case where it was desired to obtain focused images of a plurality of subjects the depths of which are different, it was necessary to capture the image of each subject in focus a plurality of times.
In recent years, a technique called light field photography has been developed, which is capable of adjusting the focus position in the later image processing (refocus) by adding a new optical element to the optical system to acquire images from multiple viewpoints.
By using this technique, there is an advantage that a failure in focusing at the time of image capturing can be made up by image processing because it is possible to perform focusing after image capturing. Further, there is also an advantage that it is possible to obtain a plurality of images focused on arbitrary subjects in an image from one captured image by changing the image processing method, and therefore, it is possible to reduce the number of times of image capturing.
In light field photography, from data of images captured from a plurality of viewpoints, the direction and intensity of a light ray that passes through each position (light field) in a plurality of positions in the space are calculated. Then, by using the information of the obtained light field, an image on the assumption that light passes through a virtual optical system and forms the image on a virtual sensor is calculated. By appropriately setting such a virtual optical system and a virtual sensor, refocus described previously is enabled. As an image capturing apparatus for acquiring the light field, a plenoptic camera (for example, Japanese Patent Laid-Open No. 2009-124213) in which a microlens array is arranged behind a main lens and a camera array in which compact cameras are arranged side by side are known. It is possible for both to obtain data of subject images from a plurality of viewpoints captured from different directions by one-time image capturing. In other words, it is possible to represent the light field photography as calculation of an image to be acquired by a virtual sensor under virtual optical conditions from data of images from a plurality of viewpoints.
In the following, the above-described processing to calculate an image acquired by a virtual sensor is referred to as “refocus processing”. As refocus processing, there is known a method in which acquired data of images from a plurality of viewpoints is subjected to projective transformation onto a virtual sensor, and added and averaged (for example, WO2008050904).
As a method for displaying an image having been subjected to refocus processing (hereinafter, a refocus image) while varying the focus position, mention is made of, for example, the method disclosed in United States Patent No. 2008/0131019. In this method, the user interface (UI) to adjust the focus position is prepared on the screen on which a refocus image is displayed and the focus position is varied via the UI. Further, in United States Patent No. 2008/0131019, the method is disclosed, which displays a refocus image whose focus position is adjusted to a subject which a user desires to put to a focus position and has given instructions through a screen on which the refocus image is displayed.
By the method proposed by each Patent Literature described above, it is not possible to check whether or not a target subject is included in a range in which refocus can be performed at the time of image capturing or at the time of editing a captured image, and therefore, it is difficult for a user to capture an image or to edit an image in a manner that the user intends.